Pen-mouse system

ABSTRACT

The disclosed pen-mouse is a tracking menu that tracks the position of the pen. A pen cursor that corresponds to the pen is moved about within the pen-mouse graphic by the pen and the pen-mouse remains stationary. The pen-mouse is moved when the location of the pen encounters a tracking boundary of the pen-mouse. The tracking boundary coincides with the graphic representing the mouse. While moving within the pen-mouse, the pen can select objects within the pen-mouse body, such as buttons, wheels, etc. The selection of a button or other virtual control causes a corresponding computer mouse button function to be executed. The execution is directed at any object designated by a pen-mouse tracking symbol, such as an arrow, that is part of the pen mouse graphic. The pen-mouse emulates functions or operations of a mouse including single button clicks, double button clicks, finger wheels, track balls, etc.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.11/754,019, filed May 25, 2007, now allowed, which is a continuation ofU.S. patent application Ser. No. 10/648,581, filed Oct. 15, 2003, U.S.Pat. No. 7,242,387, which is related to and claims priority to U.S.provisional application entitled Tracking Menu System having Ser. No.60/419,144, by Fitzmaurice et al, filed Oct. 18, 2002. This applicationis also related to U.S. application entitled Tracking Menu, System andMethod having Ser. No. 10/684,580, by Fitzmaurice et al, filed Oct. 15,2003 and to U.S. application entitled Pan-Zoom Tool having Ser. No.10/684,579, by Fitzmaurice et al, filed Oct. 15, 2003. The subjectmatter of these related applications is hereby incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is directed to a tracking menu that can be usedwith a pen based computer and that provides a pen user the functionalityof a mouse and, more particularly, to a pen-mouse that tracks a positionof a pen or stylus, allows pointing like a mouse and emulates thefunctionality of mouse buttons and other mouse inputs, such as a fingerrollable wheel.

2. Description of the Related Art

Stylus or pen-based computers, such as tablet (personal computers) PCsand personal digital assistants (PDAs) are becoming a popular type ofcomputing device. These devices operate somewhat differently than thetypical mouse-based computer systems. In the pen-based devices the useruses a stylus or pen to select and activate items, such as menucontrols/buttons, as well as to perform graphic functions, such asdrawing. In performing these different operations it is often the casethat the user must move the pen to a menu to select a function and thenreturn to the display area to perform the function, Because thesepen-based operations are somewhat different from traditional mouse basedoperations, it can be helpful to a new user to have a familiar paradigm,such as a mouse, that can be used in pen-based computers to performmouse type operations.

What is needed is an interface that provides a user with the look and“feel” of operation of a mouse and avoids the need to move to a menu toselect mouse functions, such as left button, right button, scroll. etc.

SUMMARY OF THE INVENTION

It is an aspect of the present invention to provide a graphical userinterface that emulates a mouse input device and is controlled by a penof a pen based computer.

It is another aspect of the present invention to provide a mouseemulation that functions as a tracking menu.

It is also an aspect of the present invention is to provide mouseemulation that does not use keyboard events/signals but does emulatesome keyboard keys (e.g., shift/alt)

The above aspects can be attained by a system that provides a pen basedcomputer user with a tool, a pen-mouse, that functions like a mouse butthat is controlled by a limited input device, such as a pen or stylus.The pen-mouse is a tracking menu that tracks the position of the pen. Apen-cursor that corresponds to the pen is allowed to move about within apen-mouse graphic. The pen-mouse is moved when the location of the pen,pen cursor or pen transducer sensed position encounters a trackingboundary of the pen-mouse. While moving within the pen-mouse the pen canselect objects within the pen-mouse such as buttons, wheels, etc. Theselection of a button or other virtual control causes a correspondingmouse button function to be executed. The execution focus is directed atany object designated by a tracking symbol, such as an arrow, that ispart of the pen-mouse graphic.

These together with other aspects and advantages which will besubsequently apparent, reside in the details of construction andoperation as more fully hereinafter described and claimed, referencebeing had to the accompanying drawings forming a part hereof, whereinlike numerals refer to like parts throughout.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a pen-mouse tracking menu according to the presentinvention.

FIG. 2 depicts a tracking border of the pen-mouse.

FIG. 3 illustrates tracking states.

FIG. 4 shows a pen cursor moving within the pen-mouse.

FIG. 5 shows the pen cursor moving the pen-mouse.

FIGS. 6 and 7 show a button being highlighted and selected.

FIGS. 8-10 depict the pen-mouse during pan tracking.

FIGS. 11 and 12 show the pen-mouse bar function highlighted andactivated.

FIGS. 13-23 show marking menu activation and double button functionactivation

FIG. 24 shows object dragging with the pen-mouse.

FIGS. 25 and 26 depict linear menu activation and selection.

FIG. 27 illustrates a tracking boundary not coinciding with the bodygraphic.

FIGS. 28 a-28 g illustrate different positions for the pen-mousetracking symbol.

FIG. 29 depicts a different pen-mouse graphic with a wheel function.

FIG. 30 shows a limited function and a limited graphic pen-mouse.

FIG. 31 shows a pen-mouse with activation safety features.

FIG. 32 shows an arrow shaped pen-mouse.

FIG. 33 shows pen-mouse tracking symbols changing according to systemstate.

FIG. 34 shows an interior tracking wall.

FIG. 35 shows a wall being used to provide track ball functionality.

FIG. 36 shows pen-mouse relative size.

FIG. 37 shows a desktop PC as hardware of the present invention.

FIG. 38 depicts a tablet PC as hardware of the present invention.

FIG. 39 shows components of the hardware of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is directed to a user interface, particularly, agraphical user interface (GUI) that has the appearance of a mouse 10 asdepicted in FIG. 1. This interface 10, which for convenience will becalled a pen-mouse, is a tracking menu where the pen-mouse 10 tracks theposition of a stylus or pen being used with a pen-based computer. Atracking menu is a graphical user interface widget that is controlled byeither a pen or mouse having an associated pen or mouse cursor where thecursors can be invisible. A tracking menu is invoked and dismissed inthe same manner as a traditional modal tool by clicking on a toolpalette or menu item. Like traditional menus, a tracking menu includes acluster of graphical buttons or controls. The pen cursor can be movedwithin the menu to select and interact with items, such as the buttons.However, unlike traditional menus, when the pen cursor crosses theexterior edge or tracking boundary of the menu, the menu is moved tokeep it under the cursor, The pen-mouse tracking menu, of the presentinvention, can track other types of input devices, such as a finger in atouch based menu, a traditional mouse and 3D input devices such asgloves. Additional details with respect to tracking menus can be foundin the related application noted above.

The pen-mouse 10 preferably includes a mouse graphic 12 that includes avisible border or edge 14 and a traditional arrow cursor or trackingsymbol 16. This pen-mouse tracking menu 10 tracks the position of thepen of the pen based computer system as will be discussed later in moredetail. By moving the pen, the cursor 16 can be positioned to point atan object in the same way that a cursor for a mouse is positioned topoint at an object. The mouse graphic 12 depicts three buttons 18(left—L), 20 (middle—M) and 22 (right—R) that are conventional typevirtual controls specifically for initiating mouse button down eventswhen activated by a pen down event with the pen tip positioned over oneof the buttons 18-22. The tracking symbol or cursor 16 behaves like atraditional tracking symbol used with a mouse where the tracking symbolcan point at (be positioned over) an object, such as a menu, andactivate the object when a mouse down event occurs. That is,conventional input mapping based on location of an input sensor orcursor are performed. As a result, when the cursor 16 is positioned overan activatable object and a pen down event occurs with the pen locatedover one of the buttons 18-22, a mouse down event for the correspondingmouse button occurs for the object at which the cursor points. In thisway, the pen of the pen-based computer system can be used to perform thesame functions as a mouse in a mouse based computer system.

The pen-mouse tracking menu 10, as depicted in FIG. 2, includes atracking menu tracking boundary 30 that allows the pen cursor 16 to movewithin the tracking menu 10 without causing the menu to pen-mouse 10 tomove with the movement of the pen. The boundary 30 is shown ascoinciding with the visible edge of the graphic 12 but need not be socoincident (see FIG. 27).

The operation of the pen-mouse tracking menu 10 of the present inventioncan be understood using a state diagram as depicted in FIG. 3. In state0 the pen is out-of-range 40 of the tracking system of the pen basedcomputer, such as a tablet PC, and the tracking menu 10 is visible.(preferably semi-transparent) and is positioned in the last place itresided before the pen moved out-of-range or in an initial position ifthe tracking menu has just been involved. The state 0 is entered fromstates 1 and 1 E when the pen is lifted from the tablet and movedout-of-range. The menu 10 resides in this last position until the pen isbrought back into range and the tablet PC begins tracking the pen.

The pen can move into tracking range 42 and enter state 1 in twodifferent ways. In the first into-range transition, the pen comes downat a position where the Up of the pen is outside of the tracking menuedge or tracking boundary 30. When tracking starts in this condition,the menu 10 is moved or repositioned 44 so that the cursor is inside themenu edge. This involves conventionally redrawing the semi-transparentmenu 10 at a position corresponding to the pen tip, cursor or sensed pentransducer position, This repositioning or redrawing can place the menu10 such that the menu 10 is moved the minimum distance to place the pencursor just within the edge of the menu 10. Or the repositioning canplace the menu at an arbitrary position under the cursor, such aspositioning the menu with the pen tip in the center of the menu. Thesecond into-range transition occurs when the pen tip comes into rangewhen it is within the boundary 30 of the tracking menu 10.

In state 1 the pen cursor moves freely about within the menu 10 and themenu 10 stays stationary. During this movement of the pen cursor withinthe menu 10, the system performs conventional operations, such ashighlighting buttons or controls over which the cursor passes bycomparing the position of the cursor to positions of the buttons.However, if the pen is lifted out-of-range the state moves to state 0(the pen tracking becomes inactive), if the pen encounters an edge as itmoves, state 1 E is entered and if the pen touches the tablet surfacestate 2 is entered.

To enter state 1 E the position of the pen cursor (or pen tip transducerposition) is conventionally compared to the position of the edges orboundary 30 of the tracking menu 10, When a match occurs, the cursor hashit the edge and the state 1 E is entered. In state 1 E, as long as thecursor is at or in contact with an edge as the cursor moves, thetracking menu (semi-transparent) is moved along with the cursor. Thatis, as the cursor is moved, the menu 10 is conventionally redrawn withthe cursor at the edge of the tracking menu. In state 1 E, if the pen islifted out-of-range the state moves to state 0, if the pen moves awayfrom an edge to reenter the interior of the menu the state moves tostate 1 and if the pen touches the tablet, state 2 is entered.

As discussed above, state 2 is entered when the pen touches 46 thetablet surface while in state 1 or state 1 E. In this state the pen isactive or activated such that it will cause some function to beperformed. In state 2 the active pen can be selecting a button, in whichcase the function of the button is performed, such as selection of anew. The tracking menu does not have to become transparent while instate 2. The controls or buttons of the present invention can bedesigned to work such that when you press on them, they show somehighlighting and the assigned function is executed only on pen-up. Thisallows the user to cancel their action by moving off of the button whilein the pressed state and thus preventing the triggering of the function.Or the active pen can be moving while under the control of a previouslyselected function, such as painting with a previously selectedpaintbrush or zooming based on a previous selection of a zoomtool/function. In state 2, the tracking menu 10 is made fullytransparent. In this state, the system can continue to reposition thefully transparent menu under the cursor or preferably the menu can beallowed to remain in it's last position as in state 0 (note the userwould not perceive a difference between these two alternatives). Whenthe pen is lifted from the tablet surface and contact with the tabletends, the tracking mode 42 is again entered and the menu 10 isrepositioned 48 depending on the as state. If the pen is lifted when theprior state is state 1 E, the pen is repositioned 48 at the last edgepoint of state 1 E. If the pen is lifted when the prior state is state1, the pen is repositioned 48 at the last interior point of state 1.

Additional details concerning tracking operations can be found in therelated application previously noted.

FIG. 4 illustrates a pen cursor or pen transducer location symbol 40moving to the left within the tracking boundary 30 of the pen-mouse 10and the pen-mouse 10 remaining stationary. That is, within the body ofthe virtual mouse 10, movement of the sensed position 40 of the inputtransducer does not cause the tracking menu 10 to move. In FIG. 5 themoving cursor 40 has encountered or hit the boundary 30 when moving tothe left and the encounter with the boundary causes the pen-mouse 10 tomove to the left. That is, movement of the tracking menu 10 occurs whenthe sensed position of input transducer 40 hits the tracking menu border30 while the input transducer is in a “tracking state” (i.e., not duringa mouse down state).

As depicted in FIG. 6, moving the input transducer 40 over the leftmouse button graphic 18 causes this sub-component to “highlight” (thefaint bar 46 at the top of the left mouse button 18). Note that theinput transducer 40 is still in the “tracking” state not a “mouse buttondown” state. A mouse down event (pen down over a mouse button) isdepicted in FIG. 7. When a “mouse button down” state occurs with theinput transducer (e.g., with a pen, the pen tip is engaged by a tappingdown of the pen onto the digitizer or tablet surface), the action of thetracking menu sub-component is executed. Tapping typically implies boththe pressing down and lifting up of a pen on a surface. Here, by mousebutton down when a stylus is involved is meant that the pen tip comes incontact with the surface. In this case, an event for the left mousebutton 18 is generated, the bar 46 can be strongly highlighted, the pendown cursor position 40 can highlighted by a visible state change, suchas the hot spot star shown in FIG. 7, to indicate that a hotspot actionis occurring at the tip of the arrow cursor 16. That is, any object,such as a menu item, at the tip of the arrow cursor or tracking symbol16 is activated.

The pen-mouse 10 of the present invention can not only provide thetraditional simple mouse tracking symbol and single mouse buttonfunctions discussed above but can also provide more complicatedfunctions as discussed below.

Positioning the input transducer 40 over the body 50 of the mouse 10 cancause a pan tracking symbol 52 to appear as depicted in FIG. 8. A pendown event within the mouse body subcomponent 50 of the tracking menu 10can cause the tracking symbol 40 to change to a more bold visual stateas shown in FIG. 9. In this condition with the pan function activated, amouse drag event (dragging the pen tip over the surface of the tablet PCor digitizer while the pen tip is touching the surface) within the mousebody 50 causes the tracking menu 10 to be dragged a distance anddirection corresponding to the transducer 40 motion as illustrated inFIG. 10.

Positioning the input transducer 40 over a bottom tab 60 of thepen-mouse 10 can causes a bar tracking symbol 62 to appear (see FIG. 11)indicating the user is over this subcomponent. Once again, a mouse downevent within the mouse bottom tab subcomponent 60 of the tracking menu10 causes the pen tracking symbol 40 to change to a more bold visualstate (see FIG. 12) and invokes the subcomponent action. In this case,the subcomponent action, as illustrated in FIG. 13 is to invoke a set ofmarking menus 70, The behavior of the menus 70 matches that of thetraditional marking menus. In FIG. 14, we see the user selecting “Alt”72 which is a “check-box” item (i.e., remains enabled until the “Alt”item is explicitly selected again to toggle the state off). A pen-upevent over the “Alt” menu item turns “Alt-lock” on and feedback isprovided on the body of the mouse (the “A” character 74 of FIG. 15). Apen-down on the bottom tab again brings up the marking menu 70 (see FIG.16). Selecting At 72 again (see FIG. 17) turns off “Alt-lock” (see FIG.18). Selecting “Middle” 76 from the marking menu 70 (see FIG. 19) canturn on a “Middle-lock” (see the bar 76 of FIG. 20 indicating thisstate) that results in always generating a corresponding middle buttonmouse down/up event when any of the virtual pen-mouse buttons areengaged for a down/up action (see FIG. 19). FIG. 20 shows both“Middle-lock” and “Alt-lock” turned on. Hovering the pen 40 over theleft button 18 in this mode (as depicted in FIG. 22) followed by apen-down on the pen-mouse 10 left mouse button 18 in this mode can causeleft mouse button 18 and middle mouse button events to be generated asdepicted by the highlighted pen cursor symbol 40 of FIG. 23. This pairof events is focused on any object at which the arrow cursor 16 ispointing. That is, initiating an event that requires a two button“click” on a mouse can be accomplished using the pen-mouse 10 of thepresent invention.

FIG. 24 depicts a left mouse button drag event in progress. In thisoperation, the user has positioned the arrow cursor 16 on top of the“Folder” icon 90 and has engaged the left mouse button 18 of thepen-mouse 10 and begun dragging the pen 40 across the tablet while thetip is touching, thereby dragging the pen-mouse that drags the folder90.

In FIG. 25 the user has engaged the right mouse button by placing thetracker symbol 40 over the right mouse button component 22 of thepen-mouse 10 and performing a down event with the input transducer. Thisresults in a pop-up menu 92 appearing, in this case to the left of thearrow cursor. FIG. 26 shows dragging the pen downward (see the movedposition of the highlighted pen tracking symbol as compared to FIG. 25)and this causes various menu items to highlight (in this case the “ResetPosition” item 94) as one would expect. That is, the menu 10 can behavelike a standard pop-up menu.

The pen-mouse of the present invention need not use the arrow as thetracking symbol and can have a tracking menu boundary or edge that doesnot coincide with the visible edge of the mouse graphic. Thesealternatives are depicted in FIG. 27 where the pen-mouse tracking symbol110 is a “hotspot” type symbol and the tracking menu border 112 is acircle much larger that the graphic 114.

The tracking symbol used with the pen-mouse, such as the arrow, can bepositioned or oriented anywhere around the mouse graphic body. Thischoice in positioning can be designated by the user or managed by thesystem. For example, the system can move the arrow toward a display edgeto allow easier selection of edge items. Examples of different positionsfor the symbol 120 are shown in FIGS. 28 a-28 g and the utility orusefulness of each position includes: 28 a—screen top edge objects, 28b—left handed users and top right screenedge objects, 28 c—right edgeobjects, 28 d—tom right edge objects, 28 e—bottom screen edge objects,28 f—bottom left edge objects, and 28 g—left edge objects.

The pen-mouse can have a number of different appearances, shapes orbody/graphic designs. FIG. 29 illustrates a pen-mouse 130 with adifferent shape and different functions including a wheel where thevirtual wheel can be dragged up/down and the same actions as would occurwhen rotating a wheel are performed. FIG. 30 depicts a pen-mouse 140where only the buttons 142 are shown, where the tracking symbol 144 is apredetermined distance from the buttons 142 and the tracking border orboundary 146 has additional space on the right. FIG. 31 illustrates apen-mouse 150 with traditional L, M, R buttons and modifier keys. Thetracking boundary 154 is depicted by a dashed line and coincides withthe mouse graphic 156. This pen-mouse 150 includes a neck 158 attachedto a base function region 60. The neck prevents the pen tracking symbol162, a hot spot in this case, from accidentally triggering the functionsof the base region 160. FIG. 32 shows a pen-mouse 170 shaped like anarrow or a dragable cursor with LMR buttons and an action hot spot atthe tip 172 of the arrow. The pen tracking symbol 172 is shown in thebody.

FIG. 33 shows a pen-mouse 180 in which traditional changeable cursorstates are shown. The tracking or action symbol 182 may also change as afunction of the system state. The tracking symbol may also not changeand rather only the system cursor changes which is attached to thetracking menu mouse body.

FIG. 34 depicts a pen-mouse 190 that includes an interior tracking wall192 jutting from the tracking boundary that coincides with of thepen-mouse graphic. When the pen tracking cursor encounters this wall 192the pen-mouse is moved similarly to the cursor encountering the exteriortracking boundary. FIG. 35 illustrates a pen mouse 200 with a trackingboundary 202 having tracking wall 204 extending into the interior andthat rotates the mouse graphic 206 when the pen cursor 208 encountersthe wall 204. This allows the pen-mouse 200 to function like a virtualtrack ball.

When the pen-mouse 10 of the present invention is displayed on a typicalpen based computer display, the pen-mouse 10 is preferably displayed ata size similar to other tools of computer display as shown in FIG. 36.When the pen-mouse 10 is moved about on the display is preferablymaintained on top of all objects in the display including menus andother persistent objects.

The hardware of the pen-mouse tracking menu system can be within adesktop PC 230 (see FIG. 37) or within a handheld device, such as atablet PC 240 (see FIG. 38) or a PDA, and includes an input transducer250 the position of which is tracked by a computer 252 that processesthe transducer positions and creates the tracking menu display that ispresented on a display 254 as shown in FIG. 39.

The system also includes permanent or removable storage, such asmagnetic and optical discs, RAM, ROM, etc. on which the process and datastructures of the present invention can be stored and distributed. Theprocesses can also be distributed via, for example, downloading over anetwork such as the Internet.

The present invention has been described with respect to using a tablettype personal computer. The present invention is also suitable for othertypes of systems including large display formats (front and rearprojection) both horizontal and vertical large displays, such aselectronic whiteboards. Other input devices than a pen can also be used.For example, an audience could be viewing a large display presentationand if a user wants to manipulate the display from theft seat, theycould use a laser pointer type device. Here the pen tip down event wouldbe generated by a dwell event or a secondary button on the laserpointer. The present invention has also been described with respect to asingle input device being used with the system. The invention is alsooperable with two or more PenMice active at the same time, driven by twoor more input devices. This allows two handed interaction techniques orcollaborative multi-user applications.

The many features and advantages of the invention are apparent from thedetailed specification and, thus, it is intended by the appended claimsto cover all such features and advantages of the invention that fallwithin the true spirit and scope of the invention. Further, sincenumerous modifications and changes will readily occur to those skilledin the art, it is not desired to limit the invention to the exactconstruction and operation illustrated and described, and accordinglyall suitable modifications and equivalents may be resorted to, fallingwithin the scope of the invention.

We claim:
 1. A computing device, comprising: a display; and a processorconfigured to generate a graphical user interface to be displayed on thedisplay, the graphical user interface including; a mouse graphic havingthe appearance of a computer mouse; at least one button controlpositioned within the graphic with a button event being produced whenactivated; a tracking symbol graphic associated with the mouse graphicand indicating event focus; and a tracking menu boundary providingtracking menu control of the interface for a pen type input device. 2.The computing device of claim 1, wherein the tracking boundary coincideswith an edge of the mouse graphic.
 3. The computing device of claim 1,wherein the tracking boundary comprises a safety neck.
 4. The computingdevice of claim 1, wherein the tracking menu control is activated by apen input event.
 5. The computing device of claim 1, wherein threebutton controls are provided and multiple button events can be emulated.6. The computing device of claim 1, wherein the interface is configuredfor dragging objects.
 7. The computing device of claim 1, wherein thetracking symbol graphic has an appearance corresponding to system state.8. The computing device of claim 1, wherein the tracking symbol graphicis configured to be positioned at various positions around the mousegraphic.
 9. The computing device of claim 1, wherein said trackingboundary comprises an interior tracking wall.
 10. The computing deviceof claim 1, further comprising one of a wheel control, a ball control, abar control, joystick, track pad, buttons, keyboard buttons, and statusindicators.
 11. The computing device of claim 1, further comprising: asecond mouse graphic having the appearance of a computer mouse; at leasta second button control positioned within the second graphic with asecond button event being produced when activated; a second trackingsymbol graphic associated with the second mouse graphic and indicatingevent focus; and a second tracking menu boundary providing tracking menucontrol of the interface for a second pen type input device.
 12. Acomputing system, comprising: a display; and a processor configured togenerate a graphical user interface to be displayed on the display, thegraphical user interface including: a mouse graphic having theappearance of a computer mouse; at least three button controlspositioned within the graphic with a button event being produced whenactivated by a stylus input event and the button controls changingappearance when activated; a tracking symbol graphic associated with themouse graphic, indicating event focus and system state; and a trackingmenu boundary coinciding with an edge of the mouse graphic and extendinginto the mouse graphic, and providing tracking menu control of theinterface for a pen type input device.
 13. A computer-implementedmethod, comprising: producing, via a processor, a graphical userinterface to be displayed on a display and having an appearance of acomputer mouse; causing the graphical user interface to move on thedisplay in response to movements of a pen, wherein the graphical userinterface is caused to move when a cursor for the pen encounters aboundary of the graphical user interface; and causing input eventsinitiated by the pen to be interpreted as mouse events.